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ABSTRACT 



Methods and systems for performing packet loss recovery 
when transmitting compressed video over a lossy packet - 
based network include transmitting packets of compressed 
video data from a sender to a receiver. In response to 
detecting lost or erroneously received packets, the receiver 
transmits a retransmission request to the sender. In response 
to receiving the retransmission request, the sender changes 
the periodic temporal dependency distance of a frame to be 
transmitted such that the frame depends on the frame 
associated with the retransmitted packets. The receiver 
receives the retransmitted packets and restores the frame 
corresponding the retransmitted packets in a frame buffer. 
The receiver uses the restored frame to decode a frame 
transmitted after the retransmitted packets. 

37 Claims, 12 Drawing Sheets 
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METHOD AND SYSTEMS FOR DYNAMIC before display of a particular video frame. In a lossy 

HYBRID PACKET LOSS RECOVERY FOR packet-based network, such as today's Internet, where 

VIDEO TRANSMISSION OVER LOSSY packet losses and high network latency are common, recov- 

PACKET-BASED NETWORK cring lost packets before the display times of the frames 

5 associated with the packets is not always possible. 

RELATED APPLICATION INFORMATION Therefore, some repair packets might arrive after the display 

. . - Bt „ times of the associated video frames. Conventional tech- 

•Dm application lS a cont.nuatiOD-.n.part of U-S. patent n j discafd lhcse , ate j, ke(s ,„ w RESCU 

application Ser. No 09/079,621 filed May 15, 1998 now Mn use l[)e kets t0 tnQI propagatioD . Stop . 

U.S. Pat No. 6 104 757, the disclosure of which is incor- 1Q j erf0r lion k iccomplisbed by buffering dis . 

porated herein by reference m its ent.rety played frameS) festoring ^ buflfcred {nmes wheQ repajr 

TECHNICAL FIELD packets arrive, and using the buffered frames as reference 

frames for subsequent frames. 

The present invention relates generally to methods and The main benefit of RESCU is that it allows more time for 

systems for transmitting video over a lossy packet-based 15 transport-level recovery to succeed. That is, repair packets 

network. More particularly, the present invention relates to f or a frame are useful until that frame is being used as a 

methods and systems for dynamic hybrid packet loss recov- reference frame. In order to accommodate recovery delays, 

ery for video transmission over a lossy packet-based net- RESCU designates every V th frame, P being an integer, as a 

work. periodic frame. The number of frame intervals between two 

20 consecutive periodic frames is referred to as the periodic 

BACKGROUND ART temporal dependency distance (PTDD). Every frame 

Packet losses are common in a lossy packet-based between periodic frames references only its immediately 

network, such as the Internet. For example, during high- preceding periodic frame. Such frames are referred to as 

traffic time periods, about 5% to 10% packet losses over non-periodic frames. 

connections between the east and west coasts of the United 25 FIG. 1 illustrates an example of RESCU picture coding 

States, or over trans- Atlantic or trans- Pacific connections are with a PTDD value of 2 according to the algorithms 

not unusual. Since packet losses in a lossy packet-based described in the above-referenced parent application. In 

network, such as the current best effort delivery Internet, FIG. 1, blocks F 0 through F 6 represent frames. Blocks F 0 , 

cannot be avoided, applications such as Internet-based video F 2 , F 4 and F 6 are periodic frames. Blocks F lf F 3 and F 5 are 

telephony must be structured to be tolerant of packet loss. 30 non-periodic frames. As illustrated in FIG. 1, the periodic 

Unfortunately, the quality of compressed video is very temporal dependency distance is measured from the start of 

susceptible to packet loss because of motion estimation one periodic frame to the next periodic frame. In the 

employed in video compression. Motion estimation is the illustrated example, the periodic temporal dependency dis- 

process of estimating the displacement of moving objects in 35 lance is ec l ual 10 lw0 frames. 

a video sequence. Motion estimation is currently used in When packet losses occur during transmission of a peri- 
popular video compression and decompression algorithms, odic frame, repair packets can be transmitted to repair the 
commonly referred to as codecs, such as H.261, H.263, losses. The repair packets can be retransmitted packets or 
MPEG-1, MPEG-2, and MPEG-4, to remove temporal forward error correction (FEC) packets that arrive before the 
redundancy in successive video frames. The temporal redun- 40 decoding time of the next periodic frame. If the FEC or 
dancy is removed by encoding only pixel value differences retransmitted packets arrive before the decoding of the next 
between the current image and its motion-predicted image periodic frame, error propagation can be stopped by using 
reconstructed from a previously encoded image. The prcvi- the repair packets to repair the reference frame of the next 
ously encoded image is referred to as a reference frame or periodic frame. For example, in FIG. 1, if frame F 0 is 
R-frame. In these codecs, loss of packets for a particular 4S transmitted with errors, and repair packets arrive before 
video frame manifests itself not only in the reduced quality display of frame F 2 , the repair packets can be used to repair 
of the frame in which the loss occurs, but also in subsequent the buffered F 0 before F 0 is used as a reference for frame F 2 . 
frames due to propagation of distortion to the successive Accordingly, error propagation due to errors in frame F 0 will 
frames that reference, either directly or indirectly, the erro- not extend beyond frame V x 

neously received frame. This problem is referred to as the 50 Because frames do not reference non-periodic frames, 

error propagation or error spread problem. loss in non-periodic frames does not cause error propaga- 

Most of the conventional work on loss recovery focuses lion. Since no attempts are made to restore non-periodic 
on recovering packet losses using retransmission and for- frames after display of the non-periodic frames, only peri- 
ward error correction before the scheduled displayed times odic frames being recovered need to be buffered for future 
of the video frames contained in the lost packets. However, 55 reference by succeeding periodic frames. This reduces the 
this approach is ineffective for interactive video because of memory requirements for error recovery, 
the delays in detecting and repairing the losses. For example, FIGS. 2 and 3 illustrate the benefits of RESCU. In FIG, 
in order to allow time for loss detection and repair, existing 2, an error, indicated by distortion in the image of the person, 
techniques introduce delay in the frame display times. that begins in periodic frame F 0 propagates to subsequent 
Delaying the frame display times greatly impairs the effec- 60 frames F a , F 2 , and F 3 . In FIG. 3, RESCU is used with a 
tiveness of interactive video communication. periodic temporal dependency distance of 2. Accordingly, an 

The above-referenced parent U.S. patent application, error that occurs in Fj is repaired before display of frame F 2 . 

entitled "RECOVERY FROM ERROR SPREAD USING Accordingly, the error occurring in frame F, does not 

CONTINUOUS UPDATES (RESCU)," discloses methods propagate to frame F 2 or successive frames, 

and systems for preventing error spread. Unlike conven- 65 Conventional applications of RESCU explore the effec- 

tional techniques, RESCU focuses on eliminating error tiveness of RESCU using a fixed PTDD. The PTDD deter- 

propagation instead of preventing errors due to packet loss mines a deadline within which packets need to be recovered. 
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However, network conditions vary over time. For example, Accordingly, it is an object of the present invention to 

congestion, transmission latency, loss rates and available provide methods and systems for dynamically adjusting a 

bandwidth frequently change. As network conditions PTDD value in" response to changing network conditions, 

change, the effectiveness of transport level recovery, i.e., It is anothcr object of the invention to provide an efficient 

retransmission and FEC, change, and thus the associated 5 mccnamsm f or \ oss rcC0 very when transmitting video over 

recovery delays change. Conventional techniques that utilize a { packet _ b ased network, 

a fixed PTDD period are incapable of adapting to changing Jr . 

network conditions. Accordingly, there exists a long-felt t Some of the objects of the invention having been stated 

need for methods and systems for adjusting the PTDD hereinabove and which are achieved in whole or in part by 

period to account for changing network conditions. the present invention, other objects will be evident as the 

tmr- Amir ixn/rvmAxi description proceeds, when taken in connection with the 

DISCLOSURE OF THE INVENTION accompanying drawings as best described hereinbelow. 
The methods and systems for performing packet loss 

recovery for video transmission over a lossy packet-based BRIEF DESCRIPTION OF THE DRAWINGS 

network according to the present invention include varying dcscr ip tion of preferred embodiments of the present 

the PTDD value in response to network conditions to allow is . . r r _ 

sufficient time for packets to be delivered before recovery of «?« now P"™* Wlth referetlce 10 lhe accom P a ' 

succeeding periodic frames. The PTDD value cannot be set drawings of which: 

arbitrarily large because this reduces compression efficiency. FIG. i is a block diagram illustrating RESCU picture 

Thus, finding the minimum PTDD value under given net- coding with a PTDD value of 2; * 

work conditions that maximizes periodic frame recovery is 2 o FIG. 2 is a computer-generated image of successive video 

an important aspect of the invention. frames illustrating error propagation; 

Packet loss rate, loss burst length, and transmission delays FIG. 3 is a computer-generated image of successive video 

play an important role in determining PTDD. For example, f ram es illustrating the prevention of error propagation using 

if the loss rate increases, then additional FEC packets or RESCU; 

retransmission attempts are required to maintain high loss 2 5 pjQ 4 ^ a block diagram of a system architecture on 

recovery probabilities. Thus, the PTDD requires extension M embodiments of the t invention can reside; 
to accommodate time required for the increased number ot 

repair attempts. Loss buret characteristics, i.e., the difference J^S is a block diagram illustrating error recovery using 

in time between successive losses, can also affect the PTDD. RESCU; 

For example, as network traffic undergoes an increased 30 FIG. 6 is a block diagram illustrating error recovery using 

number of burst losses, retransmission becomes more effec- RESCU with forward error correction; 

tive than forward error correction, since repair packets are FIG. 7 is a block diagram illustrating methods and sys- 

transmilted only when losses occur. Hence, loss burst char- terns for performing lazy hybrid RESCU according to an 

acteristics can influence the decision to use forward error embodiment of the present invention; 

correction, retransmission, or both for recovery. When 35 FIG. 8 is a block diagram illustrating a system for 

retransmission is used, the PTDD must be at least as long as evaluating the performance of lazy hybrid RESCU and other 

one round-trip time, i.e., the time for a packet to travel from algorithms according to an embodiment of the present - 

the sender to the receiver and back. When forward error invention; 

correction is used, the PTDD should be at least as long as the RG 9 is a graph of average pgNR versus percentage loss 
product of the time interval between two consecutive FEC 40 rate under lazy hy5rid RESCU, H.261, lntra-H.261, and RPS 
packets and the number of FEC packets required for pro- for a video 5^^^ entitled "container"; 
tecting a periodic frame. When a hybrid technique combin- Ra 1Q fa a g ^ of bft fate yereus percentagc loss rate 
ing FEC and retransmission is used, finding the minimum under d RESCUf pr 261, lntra-H.261, and RPS for 
PTDD value becomes even more complex. The present (he c entitled "container- 
invention includes mcthods aod ^c^ fo^ an 45 psNR 
optimal P I 1)1) period ,„ all of these situations. « P RESCU, H.261, Intra-H.261, and 

According to one aspect, the present invention includes a R?$ fof videQ £ ^ „ news „ ; 

dynamic algorithm for hybnd loss recovery, referred to M „ . ; e L - .1 

herein as lazy hybnd RESCU, Based on current network ™- 12 * a 

conditions, this scheme determines: (1) when to retransmit 50 under lazy hybrid RESCU H.261 Intra-H.261, and RPS for 

a lost packet, (2) when and how many FEC packets to » ^™ «qu«cc entitled news ; 

transmit, and (3) the length of the PTDD suitable to achieve FIG. 13 is a graph of average PSNR versus percentage 

good compression efficiency, as well as good error resil- loss rate under lazy hybrid RESCU, H.261, Intra-H.261, and 

ience. In this algorithm, retransmission is scheduled only RPS for a video sequence entitled "children"; 

when the sender learns that ensuing FEC packets are not 55 FIG. 14 is a graph of bit rate versus percentage loss rate 

sufficient to recover from reported packet losses. When under lazy hybrid RESCU, H.261, Intra-H.261, and RPS for 

repair packets are retransmitted, the PTDD is adjusted to a video sequence entitled "children"; and 

allow these packets to arrive in time to stop error propaga- FIGS. 15-17 arc graphs of percentage loss rate per frame 

tion. This strategy allows lazy hybrid RESCU to budget only illustrating the performance of lazy hybrid RESCU under 

a small amount of bit overhead for proactive recovery, i.e., 60 varying network conditions. 

overhead due to PTDD and FEC packets. Retransmission 

requires a larger PTDD because of network delays and larger DETAILED DESCRIPTION OF THE 

bit overhead. However, in lazy hybrid RESCU, this over- INVENTION 

head is incurred only when unanticipated burst losses occur Architecture 

and proactive recovery fails. Ilus algonthm can perform 65 J 

well when packet loss characteristics illustrate a high degree FIG. 4 illustrates a system architecture on which the 

of variability, such as in today's Internet. methods and systems for performing loss recovery accord- 
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ing to the present invention can reside. In FIG. 4 a system information to adapter 414. For example, a receiver report 

includes a sender 400 and a receiver 402. Sender 400 and packet can be sent periodically, such as every 500 millisec- 

receiver 402 can each comprise a general-purpose computer, onds. The receiver report packets can be formatted in any 

such as a personal computer or a workstation. The blocks suitable format for communicating traffic statistics to the 

illustrated in FIG. 4 within sender 400 and receiver 402 can 5 sender. In a preferred embodiment, the receiver report 

comprise hardware, software, or a combination of hardware packets can be formatted according to the real time control 

and software that performs lazy ^hybrid 1 R^CUj|de^r^d proloco l (RTCP). The mean loss burst length can be added 

her^Acco^ in an application-specific extension field of an RTCP report 

^iny|^Sri^ packet 



^JiiiaToMputcrrr^dable /medium for performing lazy hybrids 
ItES^U as descr^d^f^f^ *' ^ ^^^^*ZMi» io Adapter 414 receives the receiver report packets and 

'"^S^i^'lSS^l^ can communicate with each ^If 'J* P e k rio , d one of f PTDD a *f I i,h k ms 

other over a lossy packet-based network, such as the Inter- described below. If the loss recovery scheme uses FEC, then 

net. In the illustrated embodiment, sender 400 includes a ada f ter 4,4 u musl also c « m P ute lhe ^ 

video encoder 406, a transmitter/receiver 408, and an „ F ackels 10 be l ^nsmitted during that PTDD period. This 

adapter 400. Video encoder 406 is adapted to receive an 15 ' n J? r ™ U ? * P^to^^^^". 40 * of ^er 

incoming uncompressed video stream and encode the video 400 ' wm <* cnco f es the 1*°**™ packcts ? n< ? lc 

. _ . . tA _ .„„u frame and interleaves the FEC packets with other data 

stream using an appropriate encoding or compression tech- , , , , . r ,. , t . 

nique. For example, video encoder 406 can implement ,X -\|^*2L° £ non ;Pf n ^ ,c b ™ es - 

H.261, H.263, MPEG-1, MPEG-2, MPEG-4, or other appro- „ mmcd A ° vcr ; hat ™ D P*" 0 ?/ ^ bccausc , ,he "™ 

priate compression algorithm. Transmitter/receiver 408 20 P* n °*. * ad J usted 4*™^ « "H»» *> ««wo* 

packetizes the compressed video stream received from condmons, error propagation is reduced and compression 

• m M • • * „ tt i i_ efficiency is increased over static PTDD retransmission 

receiver 402 using an appropriate transport protocol, such as *^«*«-y » mw, ' ,BW ivuouauii^uu 

real time protocol/real time control protocol (RTP/RTCP). schemes. 

Receiver 402 includes transmitter/receiver 408 capable of 25 Retransmission -Based RESCU 
communicating with transmitter/receiver 408 of sender 400. 

Accordingly, transport module 402 may also implement a In order 10 fulIv explain the methods and systems for 
suitable transport protocol, such as RTP/RTCP. Receiver 402 dynamically updating the PTDD period according to 
also includes a video decoder 410 for receiving the com- embodiments of the present invention, retransmission-based 
pressed video stream from transmitter/receiver 408 and 30 and FEC-based RESCU must first be described. Retrans- 
decompressing the video stream using an appropriate mission is the most commonly used error recovery technique 
decompression algorithm. The decompression algorithm in reliable transport. However, due to the delay in detecting 
preferably corresponds to the compression algorithm used and retransmitting lost packets, retransmission is of less 
by video encoder 406. For example, the decompression u^y for real time video transmission, 
algorithm can be implemented according to H.261, H.263, 35 Conventional techniques suggest the extension of frame 
MPEG-1, MPEG-2, MPEG-4, or other suitable algorithm. playout time to allow for the additional time required for 
Receiver 402 can notify sender 400 of packet loss in any repair using retransmission. However, playout delay 
suitable manner. For example, receiver 402 can send a severely hinders interactive video communication. In 
negative acknowledgment or a duplicate acknowledgement contrast, retransmission-based RESCU accommodates the 
for each last packet, including repair packets. In a preferred 40 retransmission delays without introducing additional play- 
embodiment, receiver 402 sends negative acknowledge- out delays by allowing packets to be repaired within a PTDD 
ments for lost packets. The negative acknowledgments are period. 

preferably sent for periodic frames only. Since non-periodic FIG. 5 illustrates error recovery using retransmission- 
frames are not used as reference frames, the loss of packets based RESCU. In FIG. 5, the upper two horizontal lines 
relating to non-periodic frames does not cause error propa- 45 represent the liming of events that occur at the sender. The 
gation and hence require retransmission. Thus, non-periodic lower two horizontal lines represent the timing of events that 
frames are preferably not recovered after their display time occur at the receiver. Referring to the uppermost horizontal 
and therefore, explicit feedback regarding loss packets is not line, three frames are transmitted from the sender to the 
necessary. Accordingly, use of the feedback channel is receiver. A frame interval is defined as the time required to 
reduced and more bandwidth is available for transmission. 50 transmit all of the packets for one frame. The second 
According to an important aspect of the invention, horizontal line represents the packets transmitted for each 
receiver 402 includes statistics gatherer/reporter 412 for frame. In the illustrated example, three packets are trans- 
gathering statistics regarding packet loss and communicat- mined for each frame. For example, packets pi, p2, and p3 
ing these statistics to adapter 414 of sender 400. For are transmitted for frame 1. 

example, statistics gatherer/reporter 412 can collect and/or 55 The arrows between the horizontal line labeled "packet 

calculate statistics on network parameters, such as the num- transmission lime" and the horizontal line labeled "arrival 

ber of packets lost and the mean loss burst length. Since time" represent the transmission of packets over a lossy 

packets preferably have unique sequence numbers, packet network. The curved line indicates that packet p3 is lost in 

losses can be delected by gaps in the sequence numbers of transit between the sender and the receiver. However, the 

received packets. The mean loss burst length can be esti- 60 receiver receives packet p4 at time tl. Because the receiver 

mated by adding all of the instances of loss bursts, including receives packet p4 without receiving packet p3, the receiver 

a single loss, observed in an interval of 500 milliseconds and knows that a loss has occurred. Accordingly, at time tl, the 

dividing the total by the number of burst loss instances, receiver sends a negative acknowledgment (NACK) to the 

including single losses, in that same interval. The fraction of sender. The sender receives the negative acknowledgment at 

packcts lost since the last interval can also be calculated. 65 time l2 and retransmits packet p3. Packet p3 arrives at the 

Statistics gatherer/reporter 412 can periodically send receiver at time t3. The retransmitted packet p3 is used to 

receiver report packets containing the gathered statistical repair frame 1 stored in a frame buffer at the receiver. In this 
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example, the PTDD value is set to two frame intervals. 
Accordingly, frame 1 is a reference frame for frame 3. Since 
frame 1 is repaired before it is used to decode frame 3, frame 
3 can be displayed without error. 

The main disadvantage of retransmission-based RESCU 
is that a long round-trip time can prolong error propagation. 
Since lost packets require at least one round -trip time for 
recovery, errors can propagate to at least the next RTT/6^ 
frames, where RTT is the round trip time and cyis the frame 
interval. 

FEC-Based RESCU 

FEC is an alternative scheme for error recovery suitable 
for use in network environments in which frequent feedback 
is not feasible, such as mobile networks and satellite net- 
works. Since FEC is an open -loop recovery scheme, its 
associated recovery delay can be significantly less than the 
recovery delay for retransmission. An exemplary FEC-based 
coding scheme suitable for use with embodiments of the 
present invention is linear block coding (LBC). In linear 
block coding, k source packets d lt d^, . . . d k are encoded into 
n packets. In other words, FEC requires an additional n-k 
FEC packets to be included in a data stream. The n packets 
constitute a block. The LBC decoder at the receiver can 
reconstruct the original k data packets using any k packets 
from the n packet block. Efficient (n,k) LBC encoding and 
decoding algorithms have been developed and implemented 
to achieve real time performance. For example, one software 
coder can achieve a throughput of 11 MB/s on a 133 MHz 
Pentium® Processor available from Intel Corporation. 

Even though FEC can be used to recover from packet 
losses, FEC is not effective when the losses of the original 
data packets and the losses of FEC repair packets are 
correlated. Using FEC in combination with RESCU allevi- 
ates this problem by allowing the FEC repair packets of a 
periodic frame to be dispersed over the PTDD period. FEC 
•packets can be spaced apart so that FEC. and data packet 
josses are not correlated to each other, thus reducing the 
effect of bursty losses. In addition, the FEC repair packets of 
a block can be sent A time units after the data packets, where 
A can be set to any suitable time period, such as one frame 
interval. 

FIG. 6 illustrates a packet sequence using RESCU in 
which FEC packets are transmitted after the data packets of 
the frame to which the FEC packets apply but within the 
PTDD period. For example, in FIG. 6, periodic frame z is 
transmitted first and contains five data packets. FEC packets 
indicated by the shaded blocks are transmitted after trans- 
mission of the data packets for frame z but within the PTDD 
period. More particularly, the FEC packets are transmitted 
during each frame interval within the PTDD period. That is, 
A, the distance between FEC packets is set to one frame 
interval, b r 

Although FEC-based RESCU is effective, one of the 
disadvantages of FEC is that it incurs bit overhead regard- 
less of packet losses. For example, because FEC packets are 
transmitted proactively, overhead for transmitting the FEC 
packets is incurred even when no errors occur. Therefore, 
bandwidth is wasted during error- free transmission. As 
indicated above, because retransmission only occurs in 
response to an error, bandwidth is not wasted during error- 
free transmission. Accordingly, embodiments of the present 
invention utilize a hybrid scheme including both FEC- and 
relransmission-based recovery. 

Dynamic PTDD Adjustment Protocol 

Retransmission is necessary only when actual packet 
losses in a periodic frame indicate that it is not possible to 
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recover the periodic frame using FEC packets alone. Thus, 
a hybrid scheme using retransmission and FEC can be 
reactive in nature. However, the PTDD must be set large 
enough to accommodate retransmission delays prior to 

5 retransmission. Unfortunately, it is difficult to predict when 
retransmission will occur. If the PTDD is set sufficiently 
large to handle rarely occurring transmission, then bit over- 
head due to an increased PTDD will be incurred regardless 
of whether retransmission occurs, wasting bandwidth during 

10 normal transmission. In order to solve this problem, embodi- 
ments of the present invention adjust the PTDD dynamically 
in response to a retransmission request, as will be discussed 
in more detail below. 

Retransmission is most effective when applied in a truly 

15 reactive fashion, where retransmission overhead is incurred 
only at the time of retransmission. In lazy RESCU according 
to embodiments of the present invention, the PTDD is 
adjusted for retransmission only when retransmission 
occurs. During normal transmission, only a minimal amount 

20 of proactive redundancy due to FEC packets is added. 
Accordingly, lazy RESCU according to embodiments of the 
present invention achieves advantages associated with both 
FEC-based and relransmission-based error recovery. 
The protocol for dynamically adjusting the PTDD period 

25 according to embodiments of the present invention can 
execute on the system architecture described above with 
respect to FIG. 4. For example, the protocol can be imple- 
mented by software included in adapter 414 of sender 400 
illustrated in FIG. 4. Before transmitting a periodic frame F i9 

30 adapter 414 determines the number of FEC packets, f; and 
a time interval between two FEC packets, A, required for 
recovering that periodic frame. Adapter 414 sets the next 
PTDD period to 



where f, is the number of FEC packets, A is the distance 

40 between FEC packets, and 6^ is the frame interval. 

Adapter 414 sets f, and A to account for short burst losses 
for which FEC is most effective. Methods for determining f ( - 
and A will be discussed in more detail below. Thus, during 
normal traasmission, the PTDD is set to a sufficiently small 

45 value to handle FEC recovery, i.e., short burst recovery. 
In lazy RESCU according to embodiments of the present 
invention, retransmission occurs only when the sender 
learns through feedback from the receiver that ensuing FEC 
packets are not sufficient to recover lost packets for particu- 

50 lar frame. 'ITie failure to recover a sufficient number of 
packets will most likely be caused by long burst losses. 
However, since the PTDD is short, retransmitted packets 
may not arrive before the decoding and display of the next 
scheduled periodic frame. Thus, the sender needs to make an 

55 adjustment to the PTDD before retransmission. 

The main idea of lazy RESCU according to the present 
embodiment is that when transmission occurs for an already 
transmitted periodic frame F ( , the next frame F ; encoded 
after that retransmission uses F, as its reference frame 

60 instead of the immediately preceding periodic frame F*, and 
F y becomes the new periodic frame. 

FIG. 7 illustrates the adjustment of the PTDD by adapter 
414 of sender 400. In FIG. 7, frames F 0 -F fi are transmitted 
from sender 400 to receiver 402. F 0 , F 3 , and F 5 are periodic 

65 frames. F 1( F 2 , F 4 , and F 6 are non-periodic frames. Thus, 
initially, periodic frame F 3 temporally references periodic 
frame F 0 and periodic frame F 5 references periodic frame F 3 . 
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In the example, a packet loss occurs in the transmission of 
periodic frame F 0 . Before the display time of frame F 5 at 
receiver 402, receiver 402 requests and receives lost packets 
from periodic frame F 0 . Upon receiving the retransmission 
request from receiver 402, sender 400 adjusts the PTDD for 
F 5 so that F 5 temporally references frame F 0 instead of frame 
F 3 . Sender 400 then sends F 5 to the receiver. Receiver 402 
decodes frame F 5 using repaired frame F 0 stored in the frame 
buffer. Accordingly, error propagation that occurred from 
frames F 0 -F 4 does not continue to frame F 5 . 10 

If the time difference between F, and F y becomes too large, 
then temporal redundancy between the two frames is mini- 
mal. This decreases compression efficiency. As a result, a 
limit is preferably set on the lime difference for using motion 
compensated coding. This time difference, referred to herein is 
as PTDD ma;r , can be set to any suitable value, such as one 
second. If the difference is larger than this time limit, then 
Fy is intra-coded. 

The transmitter maintains a counter c,- for each periodic 
frame F, to record the number of NACKs received for F,. 20 
Retransmission of lost data packets of F, occurs only when 
all of the following conditions are met: (1) c, is greater than 
f,, the number of FEC packets transmitted for the frame, (2) 
F, was transmitted less than ?TDD max1 the maximum tem- 
poral dependency distance earlier, and (3) if F^^ is the 25 
intra-frame or non-periodic sent most recently, F, was trans- 
mitted after F^ fl and no packets of periodic frames sent 
between F^,^ and F,- are retransmitted. 

In this scheme, since the periodic frames to be referenced 
are determined by feedback from the receiver, the encoder 30 
must store all the periodic frames transmitted within the 
maximum PTDD period allowed by the system (ptdd,^. 
Since feedback can be lost, the decoder at receiver 402 must 
store any damaged periodic frames received within the 
Vtid max period. 35 

Computing f ( and A 

The number of FEC packets f ( transmitted for a given 
frame and the distance A between FEC packets are com- 
puted based on the latest "short-burst" loss characteristics. 40 
The short-burst loss characteristics are defined to be the 
mean loss rate and mean burst length of packet losses that 
appear in a loss burst involving less than 4 consecutive 
packet losses. The mean loss rate and burst length are 
computed using a weighted moving average of the sampled 45 
data. The reason for using only short-burst characteristics for 
computing f, and A is because FEC is effective only when 
packet losses are uncorrelated. When packet losses occur in 
long bursts, retransmission can be a more effective recovery 
method. Since it is difficult to predict when long burst losses 50 
can occur, the use of FEC to protect against such long burst 
losses is ineffective as it incurs unnecessarily high bit 
overhead during a quiescent period. 

Using the short-burst loss characteristics, A can be com- 
puted as described above, f, is computed according to the 55 
following algorithm. For illustration, it is assumed that the 
periodic frame F, consists of k data packets. The sender adds 



f, FEC packets such that the number of packets expected to 
be received at the receiver, EX (k,f r ), is at least equal to k (so 
that recovery is possible through FEC alone). When f. FEC 
packets are added to protect k packets of the periodic frame, 
the expected number of packets received is computed as 
follows: 



EX tk . /■ ) = £ {i x D{A, /)} + £ \j x P{fh 



P(f (V ) denotes the probability of receiving exactly j packets 
out of f, FEC packets, whose losses are assumed to be 
uncorrelated. P(f^) can be computed using a (1 -state) Ber- 
noulli model as 

By adding just enough FEC to protect against expected 
losses, unnecessary FEC overhead is minimized. In addition, 
most losses are likely to be repaired by FEC alone. When the 
estimate of FEC packets is not sufficient to recover lost 
packets, retransmission can be used to augment FEC in the 
recovery process. 

Experimental Results 

Loss recovery schemes disclosed herein improve video 
quality under lossy Internet environments by focusing on 
removing error propagation associated with motion- 
compensated video coding. To evaluate the effectiveness of 
these schemes, performance of these schemes is measured 
under varying network conditions produced by actual Inter- 
net traces. The performance is then compared to the perfor- 
mance of existing solutions, such as RPS and Intra-H.261, 
which address the error propagation problem. In the perfor- 
mance evaluation, three H.263 anchor video sequences 
produced by an 1 1.263 codec available from Telenor, one for 
each of MP EG -4 class A, B and E tests, are compared. The 
three video sequences are described in Table 1 . 

TABLE 1 





Test Video Sequences 


Video Sequence 


Class 


Description 


container 


A 


Low spatial detail and low amount 






of motion 


News 


B 


Medium spatial detail and medium 






amount of motion 


Children 


C 


Hybrid natural and synthetic 






movements 



TABLE 2 



Bit rates per frame of RESCU as PTDD increases using container video sequence 
PTDD l(bits/f) 2(%) 3(%) 4(%) 5(%) 6(%) 7(%) 8(%) 9(%) 10(%) 



RESCU + H.261 18880 6 10 14 18 20 23 26 29 32 
Periodic frame 18880 12 28 32 49 50 64 67 79 81 
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TABLE 2-continued 

Bit rates per frame of RESCU as PTDD increases using container video sequence 
PTDD 1 (bits/0 2(%) 3(%) 4(%) 5{%) 6(%) 7(%) 8(%) 9(%) 10(%) 

Intra frame 88360 N/A N/A N/A N/A N/A N/A N/A N/A N/A 



TABLE 3 



Bit rates per frame of RESCU as 


PTDD 


increases usuir 


news video sequence 




PTDD 


1 (biu/0 W 3(%) 


4(%) 


5(%) 6(%) 


7(%) 8(%) 9r» 


10(%) 


RESCU + H.261 
Periodic frame 
Intra frame 


19344 10 19 
19344 26 47 
90760 N/A N/A 


26 
61 
N/A 


33 36 
76 85 
N/A N/A 


42 46 52 
96 103 110 
N/A N/A N/A 


52 
119 
N/A 



TABLE 4 



Bit rates per frame of RESCU as PTDD increases using children video sequence 
PTDD l(bits/f) 2(%) 3(%) 4<%) 5(%) 6(%) 7(%) 8(%) 9(%) 10(%) 



RESCU + H.261 34856 12 20 26 30 38 37 42 50 52 
Periodic frame 34856 25 41 51 5& 70 75 81 90 93 
Intra frame 111600 N/A N/A N/A N/A N/A N/A N/A N/A N/A 



TABLE 5 

Average statistics for 10 different transmission traces 



No. Avg. Packet Loss Rate (%) Avg. RTT (ms) 



1 


0.5 


245 


2 


5.03 


" ■ 268 


3 


8.20 


255 


4 


8.41 


261 


5 


8.91 


355 


6 


12.23 


313 


7 


13.84 


276 


8 


14.04 


800 


9 


15.35 


572 


10 


17.45 


976 



Compression Efficiency of RESCU 

Tables 2, 3, and 4 show the percentage increase of average 
bit rate per frame or each video sequence as PTDD increases 
when RESCU is combined with H.261 (RESCU+H.261). 
When PTDD is set to one, the average bit rates of RESCU+ 
H.261 are the same as the average bit rate of H.261. The 
tables also show the average bit rate increase of periodic 
frames in RESCU+H.261 (denoted "Periodic frame" in the 
tables), and the average bit rate when every frame was 
intra -coded (denoted "Intra frame" in the tables). The results 
indicate that for each increment of PTDD, the compression 
efficiency of RESCU drops about 3% to 5% in the container 
video sequence, and about 2% to 12% in the news and 
children video sequence. From the tables, it can be seen that 
when more motion is present, the bit overhead of RESCU 
increases. The results discussed below with regard to trace 
simulations illustrate that RESCU achieves the best quality 
and bit rate tradeoffs compared to all the techniques tested. 
In addition, since the bit overhead of periodic frames is 
much less than I-frames, exploiting temporal redundancy 
between two periodic frames rather than coding the periodic 
frames as I-frames is advantageous. 



30 Simulation Setup Internet Transmission 

To emulate the loss behavior encountered in the Internet, 
12 minute video transmissions were collected over a trans- 
pacific connection every hour for a two-week period. The 
frame rate was set to 10 frames per second. Full-search 

35 motion estimation and the image size of CIF (352x288 
color) were used for all experiments. The default quantiza- 
tion step of 8 was used. A video frame was first compressed 
using a RESCU codec, which was built using an implemen- 
tation of H.261. The video frame was then packetized into 

40 approximately 256-byte packets, such that the individual 
packets contain an integral number of macroblocks. 

A wide range of round-trip times (RTT) (from about 250 
ms to over 1000 ms), and loss rates (from 0.5% to 18%) were 

4S observed. The mean loss burst length is less than 3. Most of 
loss bursts are short. Occasionally, long loss bursts involving 
more than 100 packets are also seen. Out of about 200 traces 
obtained, 10 representative traces covering a spectrum of 
(mean) loss rate and round trip time were selected. Table 5 
summarizes the average traffic characteristics observed in 
the selected transmission traces. 

Trace-Driven Simulation 

The profile information of each trace, which consists of 
55 statistics on the loss rate, round-trip delays, and the instances 
of loss bursts of lengths from 1 to over 200 observed for 
every non-overlapping 10 second segment was extracted. 
Each trace yields 72 pieces of profile information to form 
one error model for a transmission simulation experiment. 
60 Each error model is applied to construct a UCB/VINT 
network simulator (ns) setup. In the simulator, an error 
model obtained from a trace controls transmission latency 
and the number of packets being dropped for a simulated 
10-second period to follow the profile information of the 
65 corresponding 10-second period in that trace. 

Video codecs (RPS, RESCU and Inlra-H.261) built by 
modifying an implementation of H.261, and the error mod- 
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els of the selected traces are integrated with UCB/VINT 
network simulator ns. FIG. 8 illustrates the simulation setup 
which implements the system architecture described above. 
In FIG. 8, transmitter 408 of sender 400 packctizes com- 
pressed video frames and passes the packet ized sequence to 5 
network simulator ns to produce packet drops and transmis- 
sion latency. At the receiving end, a trace is generated which 
reccfrds all the received packets and their received time. The 
output trace is analyzed to measure the end video quality 
using an off-line decoder. Simulation was conducted in 10 ns 10 
runs, each with a different error model. 

Results of Performance Comparison 

In this section, the results of performance comparison of 
hybrid RESCU, FEC, RETX, RPS, H.261, and Intra-H.261 15 
are reported. Since the performance of In tra-1 12.61 depends 
on the available bandwidth, the performance of lntra-H.261 
is tested at a' bandwidth matching approximately the maxi- 
mum bandwidth used by any of the dynamic hybrid RESCU 
schemes. Results of the comparison are presented in the 20 
following order for each video sequence. 

Comparison of the performance of lazy hybrid RESCU 
(lazy) with that of FEC-only RESCU (FEC) and, 
retransmission-only RESCU (RETX). ^ 
Comparison of performance of lazy with that of exhaus- 
tive hybrid RESCU (exhaustive). 
Comparison of the performance of lazy with that of 

lntra-H.261, RPS, and H.261. 
In the next section, the results for the three video 30 
sequences mentioned and observations are discussed in 
detail. 

Comparison with Other Techniques 

FIGS. 9 and 10 show the average PSNR and bit rate at 35 
different loss rates under H.261, RPS, lntra-H.261 and lazy 
RESCU according to embodiments of the present invention 
for the container video sequence. As illustrated in FIG. 9, the 
average PSNR of H.261 falls rapidly (to as much as 12 dB 
lower) as loss rate increases because of error propagation. 40 
Even at the lowest loss rate, the lower quality of lntra-H.261 
for this loss rate is because for a target bit rate there is a limit 
on the maximum video quality it can sustain, its average 
PSNR is lower than that under RPS and lazy. This shows that 
even a few packet losses can affect H.261. intra-H.261 45 
shows a linear degradation in quality as loss rate increases 
with significantly higher bit rate than those of other schemes. 
RPS on the other hand can quickly recover from packet 
losses when RTT is small enough. However, at higher loss 
rates, only a small number of frames are received correctly 50 
and only the correctly received frames are used as reference 
frames. Even in low motion video sequences, such as the 
container sequence, motion-prediction is effective only up to 
a certain time after that frame. This is the reason for high bit 
rate RPS at high loss rates and small RTT. When the RTT is 55 
long, RPS shows poorer error resilience resulting in its 
average PSNR up to 4 dB lower than that of lazy RESCU. 
Lazy RESCU consistently gives better average PSNR than 
the other three schemes and lower bit rates than RPS and 
Intra-H.261. 60 

FIGS. 11 and 12 illustrate the performanceof H.261, RPS, 
Intra-H.261 and lazy RESCU. Lazy RESCU shows better 
average PSNR than all the other schemes. The average 
PSNR of H.261 decreases rapidly as the loss rate increases. 
Lazy RESCU shows up to 2 dB higher PSNR than that under 65 
RPS, but the bit rate is much lower in all but the instances 
with high RTT. Intra-H.261 shows 2-3 dB lower PSNR in 



,054 Bl 

14 

spite of 35 < fr-40% higher bit rate. The bit rate of lazy and 
RPS in news has increased quite a bit (about 10%) from that 
in the container sequence. This is because these techniques 
allow the lime distance between a frame and its temporally 
dependent frame to be larger than one frame interval, and 
thus more motion present in the input video sequence 
significantly reduces compression efficiency. However, for 
video sequences with a medium degree of motion such as 
news, lazy RESCU still maintains the highest video quality 
with only a small amount of bit overhead even under high 
loss rates. 

FIGS. 13 and 14 show the performance of H.261, RPS, 
Intra-H.261, and lazy RESCU for the children video 
sequence. As expected, the average PSNR decreases rapidly 
in H.261 as the loss rate increases. Despite much steeper 
drop in video quality than in the news and container 
sequences, lazy RESCU still shows a better performance 
both in terms of the average PSNR and bit rate when 
compared to RPS and lntra-H.261. 

Adaptiveness of Hybrid RESCU 

To illustrate the adaptiveness of lazy hybrid RESCU, the 
scheme is executed over a trace where network traffic shows 
high variations. In FIGS. 15-17, the results of the experi- 
ment performed using the container sequence are plotted. 
FIG. 15 shows the average PSNR over every 5-frame period 
(i.e., every half-second period), FIG. 16 shows the loss 
percentage of each frame, and FIG. 17 shows the average bit 
rate over every 5-frame period. 

The adaptiveness of lazy hybrid RESCU is clearly visible 
when it adapts the amount of repair overhead to maintain 
high quality under varying network conditions. From the 
first graph, it can be observed that that the video quality of 
the RESCU scheme drops when packet loss occurs. 
However, the video quality immediately bounces back and 
generally sustains good quality with PSNRs between 35 dB 
and 40 dB. Around frames 200-450, frames 700-1350, 
frames 2200-2400, and frames 2600-2800, the trace expe- 
riences high packet losses. During these times, it can be 
observed that the bit rate increases beyond 225 kbits/sec, 
which is the result of increased repair traffic. During the 
other times, the bit rate drops to around 180 kbits/sec which 
is approximately the same bit rate as H.261 . This is because 
during heavy loss periods, FEC packets are not enough to 
recover from losses, and retransmission is used more often. 
Since during quiescent periods, retransmission does not 
occur and only a small number of FEC packets are added, 
the bit overhead drops to minimum. These results combined 
with the results presented in the previous sections indicate 
that lazy RESCU is able to adapt to varying network 
conditions to minimize bit overhead while sustaining good 
video quality. 

Increased bit rates during lossy congested period will only 
aggravate congestion. The intent of these experiments is to 
show the effect of recovery. In a practical scheme, any 
recovery scheme has to be combined with a congestion 
control mechanism. When combined with a congestion 
control mechanism, RESCU has to increase the ratio of 
repair traffic over data traffic. Since RESCU can achieve a 
very good balance between video quality and bit overhead, 
RESCU can be a recovery mechanism used with congestion 
control. However, the performance of RESCU under a 
congestion control mechanism requires further study. 

Conclusion 

Video transmission over lossy networks, such as the 
Internet, is challenging because the quality of compressed 
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video is very susceptible to packet losses, A dynamic hybrid 
loss recovery scheme as described herein reduces error 
propagation due to packet loss in video transmission over 
the Internet. By carefully combining transport level recovery 
mechanisms such as FEC and retransmission, into a hybrid 
recovery scheme, the respective strengths can be utilized. 
Also, a dynamic recovery scheme that adapts itself to the 
network characteristics can not only improve the resilience 
to packet loss but can also reduce incurred bit overhead. 

Two dynamic hybrid recovery schemes have been 
described herein: The first is exhaustive hybrid RESCU, in 
which the sender finds minimum PDTT that incurs smallest 
bit overhead, and also satisfies a desired recovery probabil- 
ity of a periodic frame. In order to be able to meet the desired 
threshold, the PTDD should be long enough to accommo- 
date anticipated repair attempts and the associated delays 
due to both FEC and retransmission. The second is lazy 
hybrid RESCU, in which the sender chooses a FIDD that 
"proactively" provides only for recovery through FEC, and 
masks out retransmission delays in a truly "reactive" 
manner, if and when retransmission occurs, without intro- 
ducing additional playout delays. 

The simulations described herein designed based on 
actual Internet transmission tests illustrate that these 
dynamic hybrid recovery schemes provide belter perfor- 
mance than existing error-resilient schemes, both in terms of 
video quality and bit rate. Even in a high motion video 
sequence when the performance gains of both the hybrid 
techniques is much attenuated, they nonetheless show better 
overall performance when compared to RPS and Intra- 
H.261. 

The integration of rate control schemes for RESCU has 
not been investigated as it is outside of the primary scope of 
the embodiments described herein. RESCU can be extended 
to incorporate rate control schemes (e.g. frame rate 
reduction, quantization step-size adjustment etc.) that can 
reduce bit rate in times of packet losses so that congestion 
is not aggravated by increased overhead of protecting video 
frames. 

It will be understood that various details of the invention 
may be changed without departing from the scope of the 
invention. Furthermore, the foregoing description is for the 
purpose of illustration only, and not for the purpose of 
limitation — the invention being defined by the claims. 

What is claimed is: 

1. A method for performing error recovery in a packetized 
video data stream transmitted over a lossy network, the 
method comprising: 

(a) encoding a plurality of frames of video data such that 
each frame depends temporally on a reference frame; 

(b) transmitting a first frame from a sender to a receiver; 

(c) after transmitting the first frame, transmitting forward 
error correction (FEC) packets for the first frame from 
the sender to the receiver; 

(d) at the receiver, detecting an error in the first frame, 
determining whether the error can be corrected using 
the FEC packets, and, in response to determining that 
the error cannot be corrected using the FEC packets, 
requesting retransmission of lost or erroneously 
received packets associated with the first frame; and 

(e) at the sender, receiving the retransmission request, 
and, in response: 

(eXO retransmitting the lost or erroneously received 
packets to the receiver; 

(e)(ii) changing the periodic temporal dependency dis- 
tance of a second frame of the plurality of frames 
such that the second frame depends temporally on 
the first frame; and 
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(e)(iii) encoding the second frame using the first frame 
as a reference frame and transmitting the second 
frame to the receiver. 

2. The method of claim 1 wherein encoding the plurality 
of frames of video data includes encoding the frames 
according to the H.261 standard for video compression. 

3. The method of claim 1 wherein encoding the plurality 
of frames of video data includes encoding the frames 
according to the H.263 standard for video compression. 

4. The method of claim 1 wherein encoding the plurality 
of frames of video data includes encoding the frames 
according to the MPEG-1 standard for video compression. 

5. The method of claim 1 wherein encoding the plurality 
of frames of video data includes encoding the frames 
according to the MPEG-2 standard for video compression. 

6. The method of claim 1 wherein encoding the plurality 
of frames of video data includes encoding the frames 
according to the MPEG-4 standard for video compression. 

7. The method of claim 1 comprising, at the receiver: 

(f) receiving the retransmitted packets; 

(g) reconstructing the first frame using the retransmitted 
packets; 

(h) receiving the second frame; and 

(i) decoding the second frame using the reconstructed first 
video frame as a reference frame. 

8. The method of claim 7 comprising, at the receiver, 
displaying the first frame before receiving the retransmitted 
packets. 

9. The method of claim 7 wherein decoding the second 
frame includes decoding the second frame according to the 
H.261 standard for video decompression. 

10. The method of claim 7 wherein decoding the second 
frame includes decoding the second frame according to the 
H.263 standard for video decompression. 

11. The method of claim 7 wherein decoding the second 
frame includes decoding the second frame according to the 
MPEG-1 standard for video decompression. - - 

12. The method of claim 7 wherein decoding the second 
frame includes decoding the second frame according to the 
MPEG-2 standard for video decompression. 

13. The method of claim 7 wherein decoding the second 
frame includes decoding the second frame according to the 
MPEG-4 standard for video decompression. 

14. The method of claim 1 wherein the first and second 
frames comprise periodic frames. 

15. The method of claim 14 comprising transmitting a 
plurality of non-periodic frames between the first and second 
frames. 

16. A method for performing error recovery in a pack- 
etized video data stream transmitted over a lossy network, 
the method comprising: 

(a) encoding a plurality of periodic frames of video data 
such that each periodic frame depends temporally on an 
immediately preceding periodic frame; 

(b) transmitting a periodic frame F„ i being an integer, 
from a sender to a receiver; 

(c) after transmitting the periodic frame F t , transmitting 
forward error correction (FEC) packets for the periodic 
frame F, from the sender to the receiver; 

(d) at the receiver, detecting an error in the periodic frame 
F lt determining whether the error can be corrected 
using the FEC packets, and, in response to determining 
that the error cannot be corrected using the FEC 
packets, requesting retransmission of lost or errone- 
ously received packets associated with the frame F ( ; 
and 

(e) at the sender, receiving the retransmission request, 
and, in response: 
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(eXO retransmitting the lost or erroneously received (a) encoding a plurality of periodic frames of video data 

packets to the receiver; such that each periodic frame depends temporally on an 

(e)(ii) changing the periodic temporal dependency dis- immediately preceding periodic frame; 

tance of a periodic frame F y , j being an integer, such (b) transmitting a periodic frame F ( , i being an integer, 

that the frame F y depends temporally on the frame F, 5 from a sender to a receiver; 

rather than the immediately preceding periodic ( c ) after transmitting the periodic frame F-, transmitting 

frame of F^; and forward error correction (FEC) packets for the periodic 

(eXiii) encoding the frame F, using the penodic frame frame F from , ne sender l0 lhe rcC eiver; 

F, as a reference frame ana* transmitting the periodic ((J) al the n&t ^ delecting aQ emjr {n ^ frame 

™ rame V l ? 1 * t re P ei ^ r ' . . tn F,, determining whether the error can be repaired using 

17. The method of claim 16 comprising: W ^ F£C ^ [d response {Q dc J miaing th * 

(f) receiving the retransmitted packets; the crror cannot be repaired using the FEC packets, 

(g) reconstructing the periodic frame F, using the retrans- requesting retransmission of lost or erroneously 
milted packets; received packets associated with the frame F-; and 

(h) receiving the periodic frame F,-; and ( e ) a t the sender, receiving the retransmission request, 

(i) decoding the periodic frame F y using the reconstructed 15 and, in response: 

frame F, as a reference frame. fcXO retransmitting the lost or erroneously received 

18. The method of claim 16 wherein requesting retrans- packets to the receiver; 

mission of the lost or erroneously received packets com- (e)(u) changing the periodic temporal dependency dis- 

priscs transmitting a negative acknowledgment (NACK) tancc of a periodic frame F /( j being an integer, such 

from the sender to the receiver. 20 that the frame F,- depends temporally on the frame F, 

19. The method of claim 17 wherein decoding the peri- rather than the immediately preceding periodic 
odic frame F, includes decoding the periodic frame F frame of Ir^ and 

according to the H.261 standard for video decompression. (e)("0 encoding the frame F ; using the periodic frame 

20. The method of claim 17 wherein decoding the peri- F < as a reference frame and transmitting the periodic 
odic frame F, includes decoding the periodic frame F, * frame F, to the receiver. 

according to the H.263 standard for video decompression. . 28 ™ c com P mer P r °g fam P roduct of claun 27 com P ris ' 

21. The method of claim 17 wherein decoding the peri- in £ : 

odic frame F, includes decoding the periodic frame F, (0 receiving the retransmitted packets; 

according to the MPEG-1 standard for video decompression. (g) reconstructing the periodic frame F ( . using the retrans- 

22. The method of claim 17 wherein decoding the peri- ^0 nutted packets; 

odic frame F y includes decoding the periodic frame F y (°) receiving the penodic frame F y ; and 

according to the MPEG-2 standard for video decompression. (0 decoding the periodic frame F y using the reconstructed 

23. The method of claim 17 wherein decoding the peri- fram e F, as a reference frame. 

odic frame F, includes decoding the periodic frame R 29. The computer program product of claim 27 wherein 

according to the MPEG-4 standard for video decompression. 35 requesting retransmission the lost or erroneously received 

24. A method for performing error recovery when trans- P ackets comprises transmitting a negative acknowledgment 
milting a compressed video data stream across a lossy (NACK) from the sender to the receiver. • . " 
packet-based network, the method comprising: 30 ^ computer program product of claim 28 wherein 

(a) transmitting a plurality of frames of compressed video d ' cod , ia S the periodic frame F, includes decoding the pen- 
data from a sender to a receiver; 40 odlc frarae F / ^cording to the H.261 standard for video 

at the sender: decompression. ^ i_ * 

« \ j . u c e a - ___ f • 31. The computer program product of claim 28 wherein 

(b) determining a number of forward error correction v . y c & _ F . 

(FEC) packets for each frame; decoding the periodic frame F y includes decoding the pen- 

( x .... t . c , ■ * *w odic frame F according to the H.263 standard for video 

(c) transmitting the forward error correction packets for " n«inw % f & 

each frame from the sender to the receiver; 45 decompression. . **« l • 

(d) monitoring the number of negative acknowledgements A 32- Hie computer program product of claim 28 wherein 
received for each frame, and, in response to determin- decoding the periodic frame F y compraes decoding the 
ing that the number of negative acknowledgements periodic frame F, according to the MPEG-1 standard for 
exceeds a predetermined value, retransmitting a pack- vidco decompression. 

ets corresponding to the negative acknowledgements; so 33, The computer program product of claim 28 wherein 
at the receiver: decoding the periodic frame F y includes decoding the pe ri- 
fe) receiving the retransmitted packets and restoring a odic frame F, according to the MPEG-2 standard for video 
frame in the frame buffer using the retransmitted pack- decompression. 

els; anc j 34. The computer program product of claim 28 wherein 
(f) using the restored frame as a reference frame for a 55 decoding the periodic frame F y includes decoding the peri- 
frame to be displayed. °dic frame lr) according to the MPEG -4 standard for video 

25. The method of claim 24 comprising at the sender, decompression. 

setting a periodic temporal dependency (PTDD) value to an 35. A computer program product comprising computer 

initial value sufficiently large to allow recovery of lost executable instructions embodied in a computer readable 

packets using the FEC packets. 60 medium for performing steps comprising: 

26. The method of claim 25 comprising, in response to (a) transmitting a plurality of frames of compressed video 
retransmitting the lost or erroneously received packets, data from a sender to a receiver; 

setting the PTDD value to a value sufficiently large to allow at the sender: 

recovery using the retransmitted packets. (b) determining a number of forward error correction 

27. A computer program product comprising computer- 65 (FEC) packets for each frame; 

executable instructions embodied in a computer-readable (c) transmitting the forward error correction packets for 

medium for performing steps comprising: each frame from the sender to the receiver; 
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(d) monitoring the number of negative acknowledgements 
received for each frame, and, in response to determin- 
ing that the number of negative acknowledgements 
exceeds a predetermined value, retransmitting the 
packets corresponding to the negative acknowledg- 
ments; 

at the receiver: 

(e) receiving the retransmitted packet and restoring a 
frame in a frame buffer using the retransmitted packets; 
and 

(f) using the restored frame as a reference frame for a 
frame to be displayed. 
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36. The computer program product of claim 35 compris- 
ing at the sender, setting a periodic temporal dependency 
(PTDD) value to an initial value sufficiently large to allow 
recovery of lost packets using the FEC packets. 

37. The computer program product of claim 36 
comprising, in response to retransmitting the lost or errone- 
ously received packets, setting the PTDD value to a value 
sufficiently large to allow recovery of lost packets using the 
retransmitted packets. 
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